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ABSTRACT 



This report summarizes the findings of the pilot 



administration of the Colorado Survey of first- and third-year teachers and 
their administrators. These surveys reflect the opinions of the teachers and 
administrators regarding the effectiveness of Colorado teacher preparation 
institutions in preparing teachers to meet performance-based standards. The 
pilot administration was designed to refine the survey and research 
techniques. The surveyed teachers were not educated under the new standards, 
and the results of this survey cannot be used to judge the current 
effectiveness of Colorado institutions of higher education in preparing 
teacher candidates to meet the new standards. A total of 2,142 surveys were 
sent to Colorado-prepared teachers who had completed their first or third 
years, and an equal number of surveys were sent to their supervisors. In all, 
517 teachers and 975 administrators responded. Results are not reported from 
one teacher education institution from which there was a very low response 
rate. The Colorado Department of Education believes that the surveys 
generated a respectable overall response rate, although low rates of return 
from a few institutions, especially those that prepare fewer teachers, are a 
concern. Survey results also indicate that the methods used to analyze the 
data are both fair and effective, and that future administrations of the 
survey will result in valid program assessments. Results of this 
administration only yield information about the relative effectiveness of 
institution to each other, but it is expected that the survey will provide 
information about the effectiveness of institutions relative to the state's 
standards once the teachers surveyed have been prepared under those 
standards . Appendixes contain the teacher and administrator surveys . Attached 
to this report is a technical report on the survey that covers design and 
development and survey analysis in more detail. (Contains 15 tables.) (SLD) 
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Results of the 

Colorado Survey of & 3^** Year Teachers 
and their Administrators 

Pilot Administration: November 2000 
By 

Gary Lichtenstein, £d.D. 

Quality Evaluation Designs 
April 2001 



Introduction 



This report summarizes the fitidings of the pilot adtninistratioii of the Colorado Suwey of 
and 3^*^ Year Teachers and their Administrators. These surveys reflect the opinions of 
teachers and school administrators regarding the effectiveness of Colorado teacher preparation 
institutions in preparing teachers to meet performance-based standards adopted by Ae Colorado 
State Board of Education (SBE) and the Colorado Commission on Higher Education (CCHE). 
Until 2004, the surveys will be administered to respondents who were prepared under a different 
set of standards. The primary purpose of this administration was to refine the survey, 
distribution lists, and methods of analyses. Because the teachers surveyed were not prepared 
under these new standards, the current results are inappropriate for making any program approval 
decisions. The results of the survey cannot be used to judge the current effectiveness of 
Colorado institutions of higher education in preparing teacher candidates to meet the newly 
adopted standards. 

Background . . 

As part of Senate Bill 154, passed in 1999, the Colorado legislature enacted provisions to 

survey educators about the quality of teacher preparation in Colorado. Specifically, teachers in 
their first and third years of teaching in Colorado are to be distributed surveys that probe their 
opinions about their teacher preparation as it relates to the SBE’s Performance-Based Standards 
for Colorado Teachers and the CCHE’s Colorado Teacher Education Performance Measures (see 
Technical Report: Legislative Authority). 

In June 2000, Quality Evaluation Designs (QED) was contracted to develop, a^inister, 
analyze, and report findings of the survey. Survey development began immediately, m 
cooperation with staff firom the Colorado Department of Education (CDE) and CCHE, and the 
Colorado Survey Committee, a group selected to represent higher education professors of 
education, professors of statistics and educational measurement 

Each survey item is based on performance standards documents: CCHE Colorado 
Teacher Education Performance Measures (7/12/2000) and SBE Performance-Based Stand^ds 
for Colorado Teachers (1/13/2000). Survey items were reviewed and revised by representatives 
firom CDE and CCHE in July 2000, and then by representatives of the Colorado Survey 
Committee in August 2000 (see Technical Report: Survey Design & Development). In 
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September and October 2000, surveys were field tested with the cooperation of teachers and 
administrators in Douglas County Re 1 school district, the University of Colorado at Denver, and 
the University of Denver. 

CDE staff used database information to 1) identify Colorado-prepared teachers who 
completed their j^eparation in spring of 1999 and 1997, and 2) create distribution lists based on 
this information in order to send surveys to teachers and their supervisors (see Technical Report: 
Teacher & Administrator Identification and Survey Piloting. Surveys were sent out in 
November 2000, along with a cover letter ejqilaining the project and the importance of returning 
the survey, and a postage paid return envelope (see Appendices A and B: Teacher and 
Administrator Surveys). To ensure integrity of the responses, each survey contained a bar code 
that was scanned into the database upon return of the surveys in order to prevent duplicated 
responses. 



in January 2001, Denver-based J-Deko Technologies, under the close supervision of Dr. 
Lichtenstein, scanned surveys into databases. Care was taken that all returned surveys were 
accounted for and that surveys were accurately scanned. Accuracy rate of the process is 99%- 
100%. The data were then analyzed by QuaUty Evaluation Designs consultant Kadriye Ercikan- 
Alper, professor of education and measurement at the University of British Columbia, who has 
extensive eiqiertise in statistics and survey analysis. Surveys were scored using a Non-linear 
Principal Components Analysis. Each sinvey response was summarized as a single score. This 
report is based on the findings firom those analyses. 



COLORADO SURVEY DEVELOPMENT TIMELINE 



Jut 

m 


le 

n 


SB99-154 Passes, au&otizing surveys to Colorado teachers & administrators 




June 

2000 


Survey development begins 






July Survey Items drafted «& checked against CDE and CCHE Performance-Based 
1 2^0 1 Standards 








Survey items reviewed & revised by Colorado Survey Committee & higher 
education professors of teacher education and measurement 



Teacher and Administrator surveys are field tested 





M Distribution list created by CDE 






Survey sent to Colorado 1** & 3^ Year Teachers (who completed 
Colorado programs in 1997 and 1999) and their supervisors 






Returned data are scaimed into databases 



Data analyzed & reported to CDE 
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Pilot Administration Results 



Response Rates 

A total of 2,142 surveys were sent to Colorado-prepared teachers who had completed 
only one year or only three years of teaching in Colorado. An equal number of surveys were 
sent to their supervisors. 

The overall response rate for teachers was 25.42%. There are a total of 15 M^er 
education institutions from which respondents had been prepared. The response rates varied by 
institution represented from a low of 1 1.90% to a hi^ of 34.82%. Among teachers, there were 
no significant differences in the response rates among first versus third year respondents. Nor 
were there significant differences between first and friird year response rates based on 
endorsement level (K-12, elementary, secondary). 

The overall response rate among administrators was 47 .94%^ Response rates varied to 
the extent that they represented teachers of Colorado teacher preparation programs from a low of 
33.33% to a high of 65.67%. Two institutions (Ft Lewis College and University of Colorado at 
Colorado Springs) evidenced greater response rates from administrators of first year teachers 
compared to third year teachers. However, since correlations between first and third year 
responses overall were so high, and because the overall response rates from these two institutions 
were not significantly different from the mean response rate from institutions overall, QED 
deemed these differences unsubstantial. 

To protect against possible response bias, first and third year response rates of each 
institution were compared with the response rate of institutions overall. Among Teacher 
Surveys, Colorado Christian University evidenced a response rate significantly below the overall 
response rate (p<.05). Accordingly, the Teacher Survey results from this institution are not 
reported. In addition, Colorado College alumni returned only 8 surveys (response rate = 24%). 
Because of the low number of surveys (less than 10), these results are not reported. Among 
Administrator Surveys, two institutions, the University of Denver and the University of Colorado 
at Boulder evidenced response rates significantly below the overall response rate. 
Administrator Survey results from these institutions are, accordingly, not reported. In the future, 
the state can initiate follow-up requests to teachers and administrators in order to boost response 
rates where necessary. 

No significant differences were found in the response rates by endorsement level between 
first year and third year teachers or their administrators. 



* Don*t Know responses among Administrator Surveys ranged as high as 12%-50% for several ite^, preventing 
large numbers of surveys from being included in the overall analysis. As a result, 37% were useable in all analyses. 
Survey design in friture administrations will result in increased number of useable responses. 
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SUR VEY RESPONSE RATES BY RECOMMENDING INSTITUTION 





Teacher Survey 


Administrator Survey 




% 


(# Returned) 


% 


(# Returned) 


Adams State College 


27% 


(20) 


50% 


(37) 


Colorado Christian University 


12% 


(5)" 


38% 


(16) 


Colorado College 


24% 


(8)" 


47% 


V / 

(16) 


Colorado State University 


25% 


(36) 


51% 


\ / 

(74) 


Fort Lewis College 


30% 


(19) 


51% 


(32) 


Mesa State College 


34% 


(27) 


57% 


V / 

(45) 


Metropolitan State College 


23% 


(83) 


45% 


1 / 

(160) 


Regis University 


32% 


(50) 


53% 


(78) 


Univ. of Colorado at Boulder 


20% 


(42) 


41% 


V' 

(85)^ 


Univ. of Colorado at Colorado Springs 


r 21% 


(14) 


66% 


(44) 


Univ. of Colorado at Denver 


26% 


(39) 


46% 




(70) 


University of Denver 


22% 


(20) 


33% 


(30T 


University of Northern Colorado 


24% 


(104) 


48% 


(209) 


University of Southern Colorado 


35% 


(39) 


54% 


(60) 


1 Western State College 


33% 


(11) 


58% 




(19) 


Overall Response Rate 


25% 





48% 


(975) 


Survey results not reported due to significantly low survey response rati 
overall 


or fewer than j 


10 responses 



Demographics of Respondents 

who responded, 301 were teachers who had completed one year of 
te^hmg, and 215 had completed three years of teaching. A total of 60/517 (12%) hold K-12 
endorsements, 265/517 (51%) hold elementary endorsements, and 192/517 (37%) hold 
secon^ endorsements. As mentioned above, no significant differences were found in the 
overall responses by endorsement level between first and third year teachers. 



Of 4e 975 administrators who responded, 547 supervised first year teachers, 428 
supervised ^d year Of these, 126/975 (13%) supervised teachers endorsed in K-12 

subjects, 478/975 (49 /o) supervised elementary teachers, and 371/975 (38%) supervised 

sewnd^ teachers. Again, no significant differences were found by endorsement level between 
first and third year teachers. 



OVERALL RESPONSE BY ENDORSEMENT LEVEL 



Endorsement 


Teachers 


Administrators 


K-12 


12% 


13% 


Elementary 


51% 


49% 


Secondary 


37% 


38% 


Total 


100% 


100% 
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Survey Analyses 

Items 6-8 on the Teacher Surveys focused on the extent to which teachers met with their 
academic advisors during their teacher preparation. The following table summarizes those 
results. Note that undergraduate advising is not reported in cases where the respondent attended 
a different institution as an undergraduate than the institution where teacher preparation was 
completed. 



FREQUENCY OF ADVISING DURING TEACHER PREPARATION*^ 





Undergraduate Major 


Education Coursework 


Never 


Less than 
once per 
year 


Once or 
more per 
year 


Never 


Less than 
once per 
year 


Once or 
mote per 
year 


Adams State College 


0 


4 


16 


0 


4 


16 


Colorado Christian University 


0 


0 


4 


0 


2 


3 


Colorado College 


0 


0 


0 


1 


1 


4 


Colorado State University 


2 


14 


11 


6 


12 


18 


Fort Lewis College 


2 


2 


5 


0 


6 


11 


, Mesa State College 


1 


7 


14 


1 


3 


23 


Metropolitan State College 


6 


23 


24 


7 


31 


42 


Regis University 


0 


4 


14 


2 


4 


44 


Univ. of Colo, at Boulder 


7 


11 


9 


7 


14 


20 


Univ. of Colo, at Colorado Springs 


2 


3 


2 


0 


0 


14 


Univ. of Colo, at Denver 


0 


4 


6 


5 


12 


21 


University of Denver 


0 


0 


0 


4 


2 


12 


University of Northem Colorado 


3 


25 


53 


22 


32 


48 


University of Southern Colorado 


2 


7 


24 


0 


7 


31 


Western State CoUege 


0 


2 


6 


0 


2 


9 


Total Re^onses 


25 


106 


188 


55 


132 


316 



*As of July 2000, CCHE Ferformance Standards require that candidates meet with dteir advisors at least once 
per year. (This requirement was rwt in effect when respondents completed their preparation programs.) 



'' Several respondents wrote in a Don 't Know response to these questions, presumably because they couldn ‘t 
remember. Therefore, institution totals may be lower in this table than the overall response totals reported 
earlier in this report. 



The Colorado Survey Committee, upon the recommendation of QED statistics 
consultants, agreed to summarize the surveys using a single number, instead of trying to analyze 
each standard assessed. In order to validly and reliably assess teachers’ and administrators’ 
perspectives on the effectiveness of recommending institutions on each performance-based 
standard, several questions would have had to be asked about each standard, which would 
lengthen the survey and reduce response rates. The survey provides much more solid 
information when all items are combined into a single score reflecting mstitutions’ overall 
success implementing performance-based standards. 

Teacher and Administrator Surveys were examined to see whether it was appropriate to 
combine the sub-sections within each survey. No significant differences were found within any 
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of the sections between frequency distributions of first versus third year teachers, or between 
administrators of first versus third year teachers. Accordingly, responses of first and third year 
teachers and therr supervisors were combined on each section of the two surveys. Significant 
correlafioM between survey sections confirmed the appropriateness of combining all sections 
mto a smgle score (see Technical Report: Survey Analyses). 

Non-linear Principal Components Analysis (NPCA) are provided in the 
tebles below. NCPA provides a summary of each survey. Surveys representing the 15 Colorado 
mstitutions are exammed as a group and a mean (average) is calculated. In these analyses, the 
mean of all mstitutions is artificiaUy set at 0, with a standard deviation of 1. 



TEACHER SURVEY RESULTS 





Number 


Average 

Score* 


Standard Deviation 


Adams State College 


20 


-.17 


!95T 


Colorado Christian Universitv 


5 


NR 


NR 


Colorado College 


8 


NR 


NR 


Colorado State Universitv 


36 


.27 


.958 


Fort Lewis College 


19 


-.10 


.842 


Mesa State College 


27 


-.31 


.735 


Metropolitan State College 


83 


-.30 


.925 


Regis University 


50 


.37 


1.174 


Univ. of Colo, at Boulder 


42 


-.04 


1.051 


Univ* of Colo, at Colorado SDrings 


14 


.69 


1.031 


Univ. of Colo, at Denver 


39 


.11 


1.207 


University of Denver 


20 


.13 


1.172 


U. Northern Colorado 


104 


P -.22 


.839 


U. Southern Colorado 


39 


.15 


1.019 


Western State 


^ 11 


.37 


1.291 



NR Data not reported due to low response. 
* Scale range is -4 to +4. 



/ rr conducted analyses to determine whether responses varied by endorsement level 
(see Technical Report: Endorsement Level Coding). Among teachers, no significant differences 
were found ^ss mstitutions by endorsement level— K-12, elementary, secondaxy--(dfr=2 
1-.1U2, P-.903). However, differences by endorsement levels within institutions were almost 
sigmficarn (dfi=23, F-2.62, p-.064). This means that significant differences may exist between 
teachers from ^rent endorsement levels within the same institution. Among administrators, 
no sigmficmt differences were found by endorsement levels overall (df=2, F=.095, p= 909). 

HowevCT, si^ficant differences did emerge by endorsement level within Mstitutions (d^l9 
r-“2.346, p= 001). * 



When institution data are broken into the three endorsement levels, the numbers are too 
small to perform meaningful analyses. In future administrations, however, we can pool 
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responses of former administrations and conduct such, analyses. Doing this will help identify 
strengths and weaknesses by endorsement level for each institution. 



ADMINISTRATOR SURVEY RESULTS 





Number 

Responding 


Average 

Score* 


Standard Deviation 


Adams State College 


36 


-.17 


.853 


Colorado Christian University 


16 


-.20 


.799 


Colorado College 


15 


.28 


.629 


Colorado State University 


74 


-.24 


.744 


Fort Lewis College 


32 


.02 


.946 


Mesa State College 


45 


.40 


1.726 


Metropolitan State College 


159 


.11 


1.079 


Regis University 


78 1 


.01 


.830 


Univ. of Colo, at Boulder 


84 


NR 


NR 


Univ. of Colo, at Colorado Springs 


44 


.10 


.956 


Univ. of Colo, at Denver 


70 


-.19 


1261 


University of Denver 


30 


NR 


NR 


U. Northern Colorado 


209 


-.01 


1.022 


U. Southern Colorado 


60 


.06 


1.096 


Western State 


19 


.36 


1.460 


HR Data not reported due to low response. 
♦ Scale range is -3 to +3. 



Interpretation 

What do these numbers mean? Again, diese data were collected on teachers prepared 
under different standards than those measured by the survey. Therefore, the survey is not yet 
valid for any kind of decision-making. However, we can answer two important questions. 

Question #1: How similar were Colorado institutions in 1997 & 1999 in preparing Colorado 
teachers for standards implemented in 2000? 

The tables on pages 6 and 7 show the distribution of mean survey scores by institution. 
We can understand these numbers by making a -simple conversion of those scores. This 
conversion . allows, us. to look . at the . profile of responses among various institutions, but does..not 
allow us to siunmarize them into a single number. This is because it is statistically invalid to 
talfp. ■ simple averages of data that are not interval. When you are counting miles, for example, the 
distance between each interval .(mile) is the same. But with data based on opinion, as these are, 
the distance between an Agree response and a Strongly Agree response is not equal to the 
distance between an Agree response and a Disagree response. The NPCA statistical analyses 
Aat generated the^ataaon^pages 6 -and 7 <an -accommodate such .^ffer^ces, but the converted 
data cannot. 
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The range of all teacher ratings from the NPCA analyses are from -4 to +4. In the table 
below, we show the percent of Teacher Survey responses that fell within that range for each 
institution. For example, we can look at Teacher Survey results for Adams State College. 
Looking at Score Range 2, the table shows that 20% of the returned surveys from Adams State 
College almnni fell between -2 and —1 on the NPCA analysis, or Score Range 2. Next we see 
that 35% of Adams State College teachers’ mean ratings on the NCPA analysis fell in Score 
Range 3 (or between -1 and 0 on the NPCA analysis). The same percentage — 35% — also fell 
within Score Range 4, or NPCA responses of 0 and 1. In the following table, percentages are 
shown for each institution and institutions overall 



RESPONSE DISTRIBUnON OF TEACHER SURVEYS* 





Score Range 




NPCA 


1 


2 


3 


4 


5 


6 


7 




Mean —> 

#Resp 1 


(belowh2) 


(-2 to-1) 


(-1 to 0) 


(0 to 1) 


a to 2) 


(2 to 3) 


(over 3) 


Adams State College 


20 




20% 


35% 


35% 


10% 






Colo. Christiaalimv.'^ 


5 


— 


— 




— 


— 


— 


— 


Colorado College^ 


8 


— 


— 


— 


— 


— 


_ 


_ 


Colorado State Univ. 


36 




14% 


25% 


42% 


17% 


3% 




Fort Lewis College 


19 




5% 


63% 


21% 


5% 


5% 




Mesa State College 


27 




15% 


63% 


15% 


4% 


4% 




Metropolitan State Col 


83 


2% 


19% 


43% 


27% 


6% 


2% 




Regis University 


50 




8% 


36% 


30% 


14% 


10% 


2% 


U C at Boulder 


42 


5% 


5% 


52% 


21% 


12% 


2% 


2% 


U C at Color. Springs 


14 






36% 


21% 


36% 


7% 




U C at Denver 


39 


3% 


18% 


36% 


21% 


15% 


8% 




University of Denver 


20 


5% 


10% 


40% 


15% 


25% 


5% 




Univ. of Northern Co 


104 


2% 


11% 


58% 


21% 


6% 


3% 




Univ. of Southern Co 


39 


3% 


8% 


35% 


39% 


8% 


8% 




Western Stale College 


11 




9% 


36% 


27% 


9% 


18% 




OVERALL 


517 


2 % 


12 % 


45 % 


26 % 


11 % 


5 % 


0% 



^Institution totals not equaling 1 00% are due to rounding error. 
^ Data not reported due to low survey response. 



The table above shows a clustering of responses at between 3-4 in the Score Range, with 
71% of all teacher responses falling within that range. There is some variation across 
institutions, but this variation is not generally dramatic. Some of the variation within institutions 
might be e^lained by different responses within endorsement programs, which can be explored 
in future administrations. Overall, however, we can see that ratings among institutions are fairly 
closely clustered, rather than spread across the entire range. 

As the table below shows, scores among administrators were less varied than those of 
teachers. The overall range of NPCA scores was from —3 to +1, with a clear majority of 
administrator ratings falling between 0 and 1. The profile of these results is similar to the results 
of past administrations of an earlier survey, where administrators overall showed higher ratings 
than teachers did and less variation. Again, as with the teacher data, ratings cluster between 
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Score Range 3 and 4, with a M^er concentration of scores in Score Range 4 (or 0 tol) on the 
NPCA analyses. 



RESPONSE DISTRIBUTION OF ADMINISTRATOR SURVEYS* 





Score Range 


NCPA 


1 


2 


3 


4 


Mean—> 


(below ”2) 


(-2 to -1) 


(-1 to 0) 


(0 to 1) 




# 

Resp. 




Adams State College 


15 


— 


13% 


33% 


53% 


Colorado Christian University 


8 


— 


— 


38% 


63% 


Colorado College 


4 


— 


— 


25% 


75% 


Colorado State University 


8 


13% 


13% 


— 


75% 


Fort Lewis College 


14 


— 


— 


50% 


50% 


Mesa State College 


15 


27% 


— 


27% 


47% 


Metropolitan State College 


67 


2% 


6% 


22% 


70% 


Regis University 


36 


— 


8% 


22% 


70% 


Univ. of Colo, at Boulder^ 


30 


— 


— 


— 


— 


Univ. of Colo, at Colorado Springs 


16 


— 


6% 


38% 


56% 


Univ. of Ck)lo. at Denver 


16 


— 


— 


29% 


71% 


University of Denver^ 


16 


— 


— 


— 


— 


U. Northern Colorado 


79 


3% 


13% 


25% 


67% 


U. Southern Colorado 


22 


— 


— 


32% 


68% 


Western State 


5 


— 


— 


60% 


40% 


OVERALL 


359 


3% 


5% 


28% 


64% 



♦Institution totals not equating 1 00% are due to rounding error. 
Data not reported due to low survey response. 



These data answer the question posed earlier: How similar were Colorado institutions in 
1997 & 1999 in preparing teachers for standards implemented in 2000? The answer is that 
Colorado teacher preparation institutions show similar profiles in terms of teachers’ and 
administrators’ ratings on current CDE and CCHE performance standards. That is, institutions 
did not vary too much in their quality of preparation relative to current standards. 

Question #2: In what areas were Colorado teacher preparation institutions strong overall and in 
what areas might they focus increased attention in order to better prepare graduates under 
SB 154 Performance Based Standards, if they haven 't already? 

QED conducted an analysis of the percent of Agree responses (Agree + Strongly Agree) 
to each survey item. For the analysis of both the Teacher and A dmini strator Surveys, we used all 
responses that fell into Score Range 3 and 4. Among Teacher Surveys, Score Range 3 and 4 
encompassed 71% of all surveys returned. Also, because responses lower than 3 and higher than 
4 tended to cancel each other out, using the percent of agree responses best represented the 
profile of results. Among Administrator Surveys, Score Ranges 3 and 4 represented 92.7% of all 
responses. Data firom both analyses are presented on the foUowing pages. 
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Table of Item Responses — Teachers 
Score Range 3 4 Only; representing 367/517 respondents (71%) 



Item 

n 


STEM 


% Agree 
(Agree + Strongly 
Agree) 


9 


The qualify of advising 1 received related to completing my undergraduate 
major was adequate. 


66% 


10 


The qualify of advismg 1 received related to completing my teacher education 
program was adequate. 


81% 


11 


Over^ the content of my teacher education courses was relevant to my field 
experiences (i.e„ observations and student teachinei 


81% 


12 


My subject matter coursework sufficiently prepaid me to teach in my 
endorsement area. 


76% 


13 


My coUege/umversify student teaching supervisor had adequate knowledge 
about K-12 classroom |H:ac1ices. 


90% 


14 


During student teaching, 1 received adequate feedback from my 
coUege/univCTsity faculfysupervispr. 


86% 


15 


My cooperating teacher (in student teaching) was a good professional role 
model. 


92% 


16 


When 1 completed my program, 1 feh prepared to teach 


89% 


17 


1 had die opportumfy to teach Colorado Model Content Standards in my field 
experiences, including student teaching. 


86% 


Durin 


^ my prepgruHon program, I received adequate preparation in: 




18 


Using rubrics for classroom assessment. 


65% 


19 


Usmg classroom assessment for improving student achievement 


66% 


20 


Preparing students for the Colorado Student Assessment Program (CSA P) 


1-3%. 


21 


Developmg assessments that measure learning in a standards-based classroom 


59% 


22 


Developing assessments that are consistent with district curriculum 


65% 


23 


Using subject matter knowledge to design elfective curriculum 


84% 


24 


Using sutqect matter knowledge in lesson planning. 


65% 


25 


Understanding theories about .classroom manapcment 


84% 


26 


Usmg a variety of classroom management techniques 


76% 


27 


Communicating effectively with students' parents/guardianfi 


59% 


28 


Practicing a variety of instructional methods. 


92% 


29 


Modilym^ instruction for gifted learners. 


52% 


30 


Modilymg instruction for English language learners 


25% 


31 


implementing individualized education plans (lEPs) 


38% 


32 


Using technology (sueh-as coiiq)uters) to m^rove stud^t learning 


64% 


33 


leaching students computing skills. 


43% 


34 


Understanding public influence over schools (including parents, business, 
advocacy groups). 


54% 


36 


Understanding school governance at foe state, district, and school levels 


52% 




QED, April 2001 



10 



Colorado 1“ & S'"* Survey Report 



Table of Item Responses — Teachers, con’t 



37 


Relative to the needs within your endorsement/teaching area, please assess 
the effectiveness of your teacher education program in preparing you in 
the following specific areas: 


% Adequate 

(Adequate + 
Good) 




a. Phonips instruction 


45% 




h. Instracting students in.spelling 


39% 




c. Teaching standard English language usage 


57% 




d. Teaching writing strategies 


56% 




e. Teaching vocabulary development 


57% 




f. Using reading assessments to plan reading instruction 


57% 




g. Instructing students in number sense 


58% 




h. Utilizing Colorado Model Content Standards in 
reading and writing foriesson^planning 


58% 




i. Using Colorado Model Content Standards in mathematics 
for lesson planning 


55% 
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Table of Item Responses — Administrators 
Score Range 3 ^4 only; Representing 333/359 Respondents (93%) 



Item 

# 


STEM 


% Agree 
(Agree + Strongly 
Agree) 




This teacher: 


/ 


7 


Has sufficient subject matter knowledge related to the initial endotsement area 


100% 


8 


ImJ)lemeHts a variety of classroom assessments. 


98% 


9 


Uses classroom assessment ■fbr'inoiprovinfi student 


9€% 


10 


Effectively prepares students for the Colorado Student Assessment Program 
(CSAT). 


96% 


11 


Develops assessments that are aligned with standards. 


98% 


12 


Develops assessments that are consistent with district cuiriculiiTn 


99% 


13 


Uses subject matter knowledge to design effective curriculum 


.9.8% 


14 


uses subject matter knowledge to enrich lessons. 


99% 


15 


Practices effective classroom management. 


96% 


16 


Communicates well with parents/guardians. 


99% 


17 


Practices a variety of instmctional methods. 


99% 


18 


Modifies instmction for gifted learners. 


92% 


19 


Modifies mstraction for English language learners. 


92% 


20 


Implements individualized education-plans flEPs). 


96% 


21 


Uses technology (such as computers) to improve student learntrig 


96% 


22 


Understands the school s Tole in ■promoting a democratic society , 


99% 


23 


Relative to the needs mthin the initial license endorsement/teaching area, 
please assess the effectiveness of the teacher in the following specific 
areas: 


Adequate 

(Adequate + 
Good) 




a. Phonics instruction 


94% 




b. Instmcting students in spelling 


95% 




c. Teaching standard English language usage 


98% 




d» 1 caching wnting 


96% 




e. Teaching vocabulary 


97% 




f. Reading instruction 


95% 




h Utilizes Colorado Model Content Standards in 
reading and writing for lesson plarming 


97% 




i. Integrates mathematics and number sense into instruction 


97% 




j. Uses Colorado Model Content Standards in mathematics 
for Ipsson planning 


95% 
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Teacher Survey data reveal a low of 13% Agree responses for item #20 {Preparing 
Students for C-SAP) to a high of 92% Agree responses for item #15 {Cooperating Teacher M>as a 
good role model). Most survey hems were rated between 50%-75% Agree responses. 

Responses related to CCHE performance standards were consistently high, with items 10-17 all 
being rated from 81%-92% Agree. The items rated lowest included; 

Preparing students for C-SAP (13% Agree) 

Modifying instruction for English language learners (25% Agree), 

Implementing vfdividualized education plans (38% A^ee) 

Teaching students computing skills (43% Agree). 

Data from administrators tell a somewhat different story. Among 93% of the 'valid 
surveys, admimstrators provided 92%-100% Agree responses to all items. This response would 
suggest that administrators beheve that teachers who completed their preparation in 1999 and 
1997 were prepared to meet performance standards implemented in 2000. However, data from 
administrators were limhed due to n high number of Don ’t Know responses, which made only 
37% of returned surveys useable for overall analyses. In future administrations, we will provide a 
means by which to include a much higher proportion of returned surveys in the analyses. 

Conclusion 



QED believes that -die 1®* and 3*^*^ year Teacher and Administrator Surveys generated a 
respectable ov^all response rate. LoW response rates from a couple of institutions, especially 
those that prepare fewer teachers, are a concern. In the future, the state may need to send out 
follow-up requests. The number and completeness of responses suggest that survey items are 
clear and that the survey, overall, is not difficult to complete. TJHD will, however, recommend 
miaor changes for future administrations. Administrator surveys will need to be modified to 
limi t missing data. We suspect that as the survey results continue to be distributed and 
publicized, response rates wiU increase. Furthermore, we believe that the response rates by 
endorsement level were good. Within institutions, preparation programs often vary by 
endorsement type, so as responses accumulate, endorsement area analyses (and therefore 
representation) will be important 



QED also beheves hat the methods used to analyze the data are both fair and effective, 
and that future administrations of the survey will result in valid program assessments. 
Correlations across the various sections are high, which suggests that the survey, indeed, 
addresses an institution’s overall effectiveness in preparing teachers on CDE and CCHE 
performance-based standards. 

AU of this speaks only to the psychometrics of the instrument and its ability to generate valid 
numbers. It is important to realize that the results generated by this administration only reflect 
institutions’ effectiveness relative to one another. In fact, policy makers are more interested in 
institutions’ effectiveness relative to CDE and CCHE performance standards themselves — and 
ri^tly so. 

To use an analogy, in a regular foot race we are most interested in the relative rankings of 
the runners. We can tell “who wins” simply by noting who comes in first, second, third, etc. But 
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teacher i^eparation is not a foot race. If anything, it’s more like a high jump, where the bar is set 
at a height that everyone is expected to clear. These survey results cannot tell us the height of 
the bar. It is up to educators and policy makers to set that height. Only by doing so can we 
know the true meaning of survey results. Ultimately, we want everyone to clear the bar. Some 
mght clear it more than others, but Colorado citizens want to be assured that those who graduate 
Colorado’s teacher education programs have met the standards created by poUcy makers. 

Therefore, standard setting is the next step. Standard setting is a process whereby 
eductors and poUcy makers determine what scores constitute acceptable results on this survey 
Certam psychometric tools can gyide diat process, but the decision is ultimately policy-based. 
Colorado is in an excellent position to undertake this process, because now there are baseline 
data; that is, data that reflect effectiveness of institutions in meeting current performance-based 
standards before those standards were implemented. We believe that the state now has an 
excellent tool for rating teachers’ and administrators’ perceptions of an institution’s overall 
effectiveness in implementing performance-based standards. Those perceptions can be described 
by a smgle number. Educators and policy makers can now make those numbers meaningful by 
drtermining the levels teacher preparation institutions must reach in order to be considered 
effective at preparing Colorado teachers in performance-based standarcjs. 



The following pages— Appendices A & B— include reproductions of the 
Colorado f* & f Year Teacher and Administrator Surveys. 

Actual surveys were formatted using different software. Surveys on the 
following pages are accurate in terms of content, and generally 
accurate in terms of formatting. Actual surv^s were more crisply 
formatted (i.e., better line spacing and bubbles lined up). —QED 
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APPENDIX A: TEACHER SURVEY 



STATE OF COLORADO 
jst ^ yd Teaclier Survey 
for the Improvement of Teacher Education 

Please complete by DECEMBER 8 . USE ENCLOSED ENVELOPE OR RETURN TO: 

Colorado Dopartmont of Education, OfRce of Professional Services — Rm 501, 201 E, Colfax Avenue, Denver, CO 80203 

Section I: Are You A 1** or 3'** Year Teacher? 

Note: It is important that ONLY teachers in their first or third years of teaching complete this survey, 

A. Please read the following 2 questions carefully and answer appropriately: 

Yes J4o 

Since receiving your teaching license, have you taught fufl-time 

or part-time for AT LEAST 1 year but not more than 1 .5 years? O O 

Since receiving your teaching license, have you taught full-time 

or part-time for AT LEAST 3 years but not more than 3.5 years? O O 

B. If you answered YES to one of the questions above, please proceed to Section II. 

OR 

If you answered NO to BOTH questions above, please RETURN this survey. 

You need not continue. 



Section II: Basic Demographic Information 

The following irformation is f^ERY IMPORTANT for assuring that your responses are interpreted 
correctly. The demographic information that appears below (and your responses on the rest of the 
survey) is for statistical purposes only and will not be released to your college or university, supervisor, 
school district or the public. 



A. Important information is summarized below. Please answer questions 1-4 based on this information: 



CDE DATABASE -INFORMATION HERE: 

1 . Recommending Institution 

2. Initial Endorsement 

3. School District employed 

4 Sp>ir»r»l r>f f»tnr»lmnnf>Tit 



1. Institution where 1 completed 


Correct 


teacher prepar^on: 


O 


2. Endorsement is: 


O 


3. School District is: 


O 


4. School is: 


O 



Incorrect Correction (if above information is Incorrect) 

O 

O 

O 

O 



B. IMPORTANT QUESTION ABOUT YOUR PREPARATION No 

5. Have you, as a licensed teacher, EVER tau^t a course in the endorsement 

area listed in #2, above? O O 

If ybu answered YES to #5, above, then please answer the following survey questions ONLY in terms of the 
endorsement area for which you were prepared in your teacher preparation program. 



if you answered NO to #5, above, please return this survey. You need not continue. 



QED 
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APPENDIX A: TEACHER SURVEY 

Section II: Questions About Your Preparation Program 

6. Did you complete your teacher preparation program (i.e., education 
coursework, student teaching) at the same institution where you 
completed your undergraduate major? 



Yes No 
O O 



7. 1 met wlh a faculty advisor in my undergraduate 
major to discuss my program progress: 

8. 1 met with a teacher education faculty 
advisor to discuss my 



Less Than 

Never Once Per Year 
O O 



Once or More 
Per Year 
O 




9. The quahly of advising 1 received related to completLag my 
undergraduate major was adequate. 


O 


O 


O 


O 


10. The quality of advising I received related to completing my 
teacher education program was adequate. 


O 


O 


O 


O 


11 . Overall, the content of my teacher education courses was relevant 
to my field experiences (i.e., observations and student teaching). 


O 


O 


O 


O 


12. My subject matter coursework sufficiently prepared me to 
teach in my endorsement area. 


O 


O 


O 


O 


13. My college/universily student teaching supervisor had 
adequate knowledge about K-12 classroom practices. 


O 


O 


O 


O 


14. During student teaching, I received adequate feedback from my 
college/university faculty supervisor. 


O 


O 


O 


O 


15. My cooperating teacher (in student teaching) 
was a good professional role model 


O 


O 


O 


O 


16. When 1 completed my program, 1 felt prepared to teach. 


O 


O 


O 


O 


17. I had the opportunity to teach Colorado Model Content 
standards in my field experiences, including student tfiarliing 


O 


O 


O 


O 


Section ni: Questions About Your Preparation 

During my teacher preparation program, 

I received adeauate vreoaration in... 


Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


1 8. Using rubrics for classroom assessment 


O 


O 


O 


O 


19. Using classroom assessment for improving student achievement 


O 


O 


O 


O 


20. Preparing students for the Colorado Student Assessment 
Program (CSAP) 


O 


O 


O 


O 


21. Developing assessments that measure learning in a standards-based 
classroom 


O 


O 


O 


O 


22. Developing assessments that are consistent with district curriculum 


O 


O 


O 


O 


23. Using subject matter knowledge to design effective curriculum 


O 


O 


O 


O 
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APPENDIX A: TEACHER SURVEY 



During my teacher preparation program, 
I received adequate preparation in . . . 


Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


24. Developing assessments that are aligned with standards 


O 


O 


O 


O 


25. Understanding theories about classroom management 


O 


O 


O 


O 


26. Using a variety of classroom management techniques 


O 


O 


O 


O 


27. Communicating effectively with students* parents/guardians 


O 


O 


O 


O 


28. Practicing a variety of instructional methods 


O 


O 


O 


O 


29. Modifying instruction for gifted learners 


O 


O 


O 


O 


During my teacher preparation program, 
I received adequate preparation in,.. 


Strongly 

Disagree 


Disagree 


Agree 


Strongly 

A^ee 


30. Modifying instniction for English language learners 


O 


O 


O 


O 


31. Implementing Individualized Education Plans (lEPs) 


O 


O 


O 


O 


32. Using technology (such as computers) to improve student learning 


O 


O 


O 


O 


33. Teaching students computing skills 


O 


O 


O 


O 


34. Understand^g public influence over schools 
(including parents, business, advocacy groups) 


O 


O 


O 


O 


35. Understanding the school's role in promoting a democratic society 


O 


O 


O 


O 


36. Understanding school governance at the state, tiistricti and school levds O 


O 


O 


O 



Section IV — ^Preparation in Specific Skills 

Note: Laws enacted in Colorado in 1 999 require that all teachers in Colorado be prepared in the following areas. 
37. Relative to Ijie needs within your endorsement/teaching area, please assess the effectiveness of your teacher 



education program in preparing you in the following specific areas: 


Very 

Poor 


Poor 


Adequate 


Good 


a. Phonics instruction 


O 


O 


O 


O 


b. Instructing students n spelling 


O 


O 


O 


O 


c. Teaching standard English language usage 


O 


O 


O 


O 


d. Teaching writing strategies 


O 


O 


O 


O 


e. Teaching vocabulary development 


O 


O 


O 


O 


f. Using reading assessments to plan reading instruction 


O 


O 


O 


O 


g. Instructing students in Number Sense 


O 


O 


O 


O 


h. Utilizing Colorado Model Content Standards in 
Reading & Writing for lesson planning 


O 


O 


O 


O 


i. Using Colorado Model Content Standards in Mathematics 
for lesson planning. 


O 


O 


O 


O 



Q '^ED 



Thank you for your time and cooperation~Good luck in your teaching! 
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APPENDIX B: ADMINISTRATOR SURVEY 



STATE OF COLORADO 
Year Administrator Survey 
for the Improvement of Teacher Education 

Please complete by DECEMBER 8 . USE ENCLOSED ENVELOPE OR RETURN TO: 

Colorado Department of Education, Office of Professional Services — Rm 501, 201 E. Colfax Avenue, Denver, CO 80203 



Section I: Basic Demographic Information 

We are interested in the quality of preparation of the following teacher. The information you provide is 
confidential and will be used only for analyses that aggregate results of all teachers from specific Colorado 
teacher preparation programs. Complete the survey ONLY for the teacher whose name appears in the box, 
and ONLY in terms of the Initial License Endorsement Area. 



CDE DATA BASE INFORMATION HERE: 

1 . Teacher Name: 

2. Initial License Endorsement Area: 

3. School District: 

4. School: 



IMPORTANT INFORMATION 

5. 1 am a supervisor for the above named teacher: 

6. 1 have supervised this teacher when he/she has taught at least one course in 
the endorsement area hsted in #2, above: 



YES 


NO 


o 


O 


o 


O 



NOTE > If you answered to either Question 5 OR Question 6, please discontinue completing this survey 

and return it to CDE in the envelope provided. Thank you. 



Section II: Questions About This Teacher’s Overall Preparation 









This teacher: 

7. Has sufBcient subject matter knowledge related to the 
initial endorsement area 



8. Implements a variety of classroom assessments 



A 






o 



o 



o 



o 



o 



o 



o 



o 



o 



o 
















9. Uses classroom assessment for improving student achievement 

10. Effectively prepares students for the Colorado Student Assessment 
Program (CSAP) 



O 


O 


o 


o 


o 


O 


O 


o 


o 


o 
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APPENDIX B: ADMINISTRATOR SURVEY 



1 1 . Develops assessments that are aligned with standards 

12. Develops assessments that arc consistent with district curriculum 

13. Uses subject matter knowledge to design efTectivc curriculum 

14. Uses subject matter knowledge to enrich lessons 

15. lYacticcs effective classroom management 

16. Communicates well with parents/guardians 

17. Practices a variety of instructional methods 

18. Modifies instruction for gifted learners 

19. Modifies instruction for English language learners 

20. Implements Individualized Education Plans (lEPs) 

21 . Uses technology (such as computers) to improve student learning 

22. Understands the school’s role in promoting a democratic society 

Section III — ^Preparation in Specific Skills 

23. Relative to the needs within the initial license endorsementiteaching area. 











mm 




Rmsm: 


'^m 






o 


o 


o 


o 


6 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 




a. Phonics instruction 

b. Instructing students in spelling 

c. Teaching standard English language usage 

d. Teaching writing 

e. Teaching vocabulary 
f Reading instruction 

h. Utilizes Colorado Model Content Standards in 
Reading & Writing for lesson planning 

i. Integrates mathematics and number sense into instruction 

j. Uses Colorado Model Content Standards in Mathematics 
for lesson planning. 









o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 


o 



Thanks Very Much For Your Time & Insights 



O 
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Colorado Survey of 1** & 3'** Year Teachers 
and their Administrators 



Technical Report 
By 

Gary Lichtenstein, Ed.D. 
Quality Evaluation Designs 
April 2001 



I. 



Legisladve Authority 

Legislative authority for the Coloiado 1’ and 3”* Year surveys to teachers and their 
^pervtsoK is granted by SBW-154. 22-60.5-1 16(2), CXS. Legislative authority for surveying 
?! Performance-Baaed Standards is found in Section 22 

0Xe)-(b), C.R.S. Authority for surveying Colorado Commission of Higher 
E^cation (CCHE) Colorado Teacher Education Performance Measures is in SB99-154 23-1- 
121 (2)(a)-(f), C.R.S. ’ 

n. Surv^ Design & Development 

Teacher and Administrator Surveys were designed based on CDE’s Performance-Based 
Stmdards for Colorado Teachers (Adopted 1/13/00) and CCHE’s Colorado Teacher Education 
Performance Measures (7/12/00). Items were created based on CDE and CCHE 
respresentatives’ ^essment of 1) the most important provisions of the standards and OED’s 
^essment of which standards could be most appropriately assessed on a survey. Items were 
oratted by QED, then revised based on extensive feedback firom CDE and CCHE staff. On 
Augist 8, 2000, QED met with the Colorado Survey Committee, comprised of CDE and CCHE 
staff as weU as representatives from Colorado institutions of higher education, including 
profe^ors of teacher education and experts in statistics and measurement Items were further 
refined based on feedback fi»m that group. Also at that meeting, important decisions were made 
regardmg ^ey design and piloting. Subsequently, drafts of the surveys were posted on an 
electromc terve, created and maintained by CDE especially for posting information about the 
Colorado ^rveys. The listserve is accessible to all Colorado Deans of Education and others 
who they designate as appropriate. In September and October 2000, versions of the survey were 
field ^ted with tochers m the Douglas County School District, and with teachers and 
admmistrnors enrolled m courses at University of Denver and the University of Colorado 

Denver The foUowmg shows the aUgnment of survey items with CDE and CCHE Performance- 
Based Standards. 



1 & 3^ Year Surv^ Coverage— Items x Standards: November 2000 Pilot Survey 



— 


“Siirvev 


^ CDE Standards 


Teacher 


Administrator 


Standard 1 : Knowledge of Literacy 






1.1 Plan & organize reading instruction based on assessment 


37f 




1.2 Phonics and linguistic skills related to reading instruction 


37 a, b 


23 a, b 


1.4 Support reading & wnting development 


37 c, d, e 


23 c, d e, f 


1.5 Use CO content standards m reading & writing for instruction 


37 h 


23 h 


Standard!: Knowledge of Mathematics 






2.1 Develop in students understanding and use of basic math skills 


37 g 


23 i 


2.2 Use CO content standards in math for insimotion 


371 


23 j 





Survey Item(s) 1 


1 CD£ Standards 


Teacher 


Administrator | 


1 Standard 3: Knowledge of Standards & Assessment 






1 3 .3 Develop and utilize a variety of informal and formal assessments 


18, 19 


8,9 i 


1 3.5 Use assessment data as a basis for standards-based instruction 


21 


11 1 


I 3.7 Prepare students for C-SAP and other achievement tests 


20 


10 


1 3.8 Ensure that instruction is consistent with CO content standards 


24 


-- 1 


1 standard 4: Knowledge of Content 






1 4.1 Utilize content knowledge to ensure student learning 


22 


12 


1 4.3 Apply expert content knowledge to enrich student learning 


23 


7, 13 1 


1 Standard 5: Kno^edge of Classroom and Instructional Management 






1 5.2 Apply sound disciplinary practices in the classroom 


25, 26 


15 


1 5.8 Communicate with parents/guardians effectively 


27 


16 1 


1 Standard 6: Knowledge of Individualized Instruction 






1 6.1 Employ a wide range of instructional techniques 


28 


17 


1 6.2 Desif>n/modify instmclion to meet needs of exception^d learners 


29,30 


18, 19 1 


1 6.5 Develop and apply individualized education plans (lEPs) 


31 


20 


1 Staodard 7: Knowledge of Technology 






1 7.2 Use technology to increase student achievement 


32 


21 j 


1 7.5 Instruct students in basic technology skiUs 


33 


1 


I Standard 8: Democracy, Educational €k>vemance & Careers in 
1 Teaching 






I 8.1 Model and articulate democratic ideal to students 


35 


22 


I 8.3 Influences on educational practices 


34, 36 


1 


1 CCHE Standards 






1 Standard b: Ongoing screening and counseling of teacher candidates by 
I practicing teachers or faculty members 






1 (1): Faculty meet at least once per year with candidates 


7, 8, 9, 10 


— 1 


1 Standard c: Course work and field experience that integrates theory, 
1 practice, and standards-based training 


11,16,17 


— 1 


1 Standard d: Candidates complete a minimum of 800 hours of field 
1 experience that relates to predetermined learning standards 






1 (3.2); Students are provided strong role models in student teaching 


13, 15 


1 


1 Standard e: Demcmstrate skills required for licensure as specified by the 
1 State Board of Education 






1 (4): Student teachers are provided continuous feedback and support 

1 finm college faculty during student teaching 


14 


8-23 


1 standard f: Assessment of candidates subject matter knowledge 






J (3): First year teacher is able to apply content knowledge in the K-12 
j classroom 


12, 23, 24 


7 1 
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III. Teacher & Administrator Identification & Survey Piloting 

The survey assesses standards that Colorado teacher preparation programs must follow in 
preparing new teachers. These standards were adopted in 2000 and teacher education programs 
were redesigned to comply with these standards in 2000-2001. Therefore, this survey currently 
assesses teachers on standards that were not in force when the teachers were enrolled in their 
preparation programs. In Spring 2004, the survey will be administered to teachers prepared 
under the performance-based standards. The table below shows the administration dates of the 
Colorado Surveys and the classes from which first and third year teachers who will complete the 
surveys graduated. 



i** & 3^^ Year Survey Administration & Corresponding Graduating Class 



Yr Teaching 



TaHluoo 



Spring 2601 



Cf^sof^D 
G l ass o f T 99 8 




















Survey respondents were identified by crossing two CDE databases; the licensure 
database, which shows the year a candidate’s license was issued and the recommending 
institution, and the human resources database, updated annually based on district-submitted 
information that details teachers’ names, number of years teaching, and current teaching 
position. Using this information, CDE generated a list of teachers to be sent surveys, and an 
identical list to the teachers’ school principals. Questions on Teacher and Administrator surveys 
confirmed respondents’ eligibility to complete the survey. Teachers who had not taught either 
one or three years were ineligible, as well as teachers who had never taught in their endorsement 
area. Administrators were ineligible if they had not supervised the first or third year teacher 
identified on the survey the administrator received. If the administrator had not supervised the 
teacher in the specific endorsement area noted on the survey, the administrator was ineligible to 
complete the survey. 

Item #6 on the Teacher Survey asks whether a candidate completed teacher preparation 
at the same institution where he or she completed his or her undergraduate major. If the answer 
was “no,” items related to undergraduate advising (#7, #9, and #12) were eliminated from the 
analyses. 



Using LabelVision 2000 software, bar codes were put on each survey. Bar codes 
identified the last four digits of the teacher’s social security number, recommending institution, 
endorsement, number of years taught (1 or 3), school district and school. Returned surveys were 
scanned using Remark OMR software to create response databases. This software captures exact 
images of each survey, flags double or ambiguous responses, then organizes data into ASCII, 
Access, and/or SPSS database files. The bar code software ensures that each respondent returned 
only one survey. 



IV. Survey Analysis 

Survey response rates were calculated based on the number of eligible surveys returned. 

Surveys were analyzed using the Non-linear Principal Components Analysis (NPCA) of 
SPSS. NPCA was deemed the optimal method for these analyses for two primary reasons. First, 
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optimal scaling (of which NCPA is a component of) is designed to be used on non-interval data. 
Second, NCPA is a more effective method than item response theory (IRT) when analyzing a 
relatively small number of 

cases, which was the case in the by-institution analyses. For more on NPCA, visit: 
www.spss.co.kr/cool/papers/optimal scaling.htm (note: use imderline ( _ ) between “optimal” 
and “scaling.” 

First, QED looked at the different sections of the Teacher and Administrator surveys to be 
sure that results were similar in each section. Initial analyses sought to confirm that correlations 
among these sections were moderate to high for both Teacher and Administrator surveys. 
Correlations between first and third year teachers were compared on each section, and then on all 
sections. No significant differences between first and third year teachers were foimd on sectional 
correlations in either Teacher or Administrator surveys. Accordingly, responses for first and 
third year teachers were combined. Inter-section correlations were hi^ and positive, as the table 
below shows. Based on these results, responses on all sections were combined into a single 
score. 



Intersection Correlation on Teacher Surv^s (n=517) 





Section n 


Section m 


Section IV 


All Sections 


Section n 


1.000 


.728** 


.435** 


.811** 


Section III 




1.000 


. 529** 


949 * ♦ 


Section IV 






1.000 


.714** 


All Sections 








1.000 


**Correlation significant at p<.01 (2-tailed) 



Intersection Correlation on Administrator Surveys (n=3S9) 





Section n 


Section in 


All Sections 


Section n 


1.000 


.663** 


.961** 


Section in 




1.000 


.837** 


All Sections 






1.000 


**Correlation significant at p<.C 


11 (2-tailed) 



V. Endorsement Level Coding 

Endorsement level was determined using the following coding. Numbers reflect CDE 
endorsement codes: 

K-12 

Art (020000) 

Physical Education (080300) 

Music (120100) 

Special Education (198120, 198130, 198160, 198170) 

Elementary 

Elementary Education (180100) 

Early Childhood Education (180101) 

Early Childhood Special Ed (191801, 198180) 

Linguistically Different (230821) 
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Secondary 

Agriculture (010000) 

Business Education (032300) 

English Language Arts (050000) 
Speech (050500) 

Theatre (050600) 

Languages (060208, 060209, 060219) 



Home Economics (090000) 
Industrial Arts (100000) 
Mathematics (110000) 
Science (130101) 

Social Studies (150000) 
Middle School (180200) 
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Publications From OJJDP 



OJJDP produces a variety of publications— 

Fact Sheets, Bulletins, Summaries, Reports, 
and the Juvenile Justice journal— along with 
videotapes, including broadcasts from the juve- 
nile justice telecommunications initiative. 
Through OJJDP’s Juvenile Justice Clearing- 
house (JJC), these publications and other re- 
sources are as close as your phone, fax, 
computer, or mailbox. 

Phone: 

800-638-8736 

(Monday-Friday, 8:30 a.m.-7;00 p.m. ET) 

Fax: 

301-519-5212 

Online: 

OJJDP Home Page: 

www.ojjdp.ncjrs.org 

E-Mail: 

puborder@ncjrs.org (to order materials) 
askncjrs@ncjrs.org (to ask questions 
about materials) 

Mail: 

Juvenile Justice Clearinghouse/NCJRS 
P.O. Box 6000, Rockville, MD 20849-6000 
Fact Sheets and Bulletins are also available 
through fax on demand. 

Fax on Demand: 

800-638-8736, select option 1, select option 2, 
and listen for instructions 
To ensure timely notice of new publications, 
subscribe to JUVJUST, OJJDP’s electronic 
mailing list. 

JUVJUST Mailing List: 

e-mail to listproc@ncjrs.org 
leave the subject line blank 
type subscribe juvjust your name 
In addition, JJC, through the National Criminal 
Justice Reference Service (NCJRS), is the 
repository for tens of thousands of criminal and 
juvenile justice publications and resources from 
around the world. They are abstracted and 
made available through a database, which is 
searchable online (www.ncjrs.org/ 
database.htm). You are also welcome to submit 
materials to JJC for inclusion in the database. 
The following list highlights popular and re- 
cently published OJJDP documents and video- 
tapes, grouped by topical areas. 

The Office of Juvenile Justice and Delinquency 
Prevention Brochure (1996, NCJ 144527 (23 
pp.)) offers more information about the agency. 

The OJJDP Publications List (BC0001 15) offers 
a complete list of OJJDP publications and is 
also available online. 

OJJDP sponsors a teleconference initiative, 
and a flyer (LT 1 1 6) offers a complete list of 
videos available from these broadcasts. 

Corrections and Detention 

Beyond the Walls: Improving Conditions of 
Confinement for Youth in Custody. 1998, 

NCJ 164727 (116 pp.). 

Boot Camps for Juvenile Offenders. 1 997, 

NCJ 164258 (42 pp.). 

Disproportionate Minority Confinement: 1997 
Update. 1998, NCJ 170606 (12 pp.). 

Juvenile Arrests 1996. 1997, NCJ 167578 
(12 pp.). 

Juvenile Court Statistics 1995. 1998, 

NCJ 170607(112 pp.). 



Courts 

Offenders in Juvenile Court, 1995. 1997, 

NCJ 167885 (12 pp.). 

RESTTA National Directory of Restitution 
and Community Service Programs. 1998, 

NCJ 166365 (500 pp.), $33.50. 

Youth Courts: A National Movement Telecon- 
ference {V\6eo). 1998, NCJ 171149 (120 min.), 
$17.00. 

Delinquency Prevention 

1997 Report to Congress: Title V Incentive 
Grants for Local Delinquency Prevention 
Programs. 1998, NCJ 170605 (71 pp.). 
Allegheny County, PA: Mobilizing To Reduce 
Juvenile Crime. 1997, NCJ 165693 (12 pp.). 
Combating Violence and Delinquency: The 
National Juvenile Justice Action Plan (Report). 
1996, NCJ 157106 (200 pp.). 

Combating Violence and Delinquency: The 
National Juvenile Justice Action Plan (Sum- 
mary). 1996, NCJ 157105 (36 pp.). 

Mentoring— A Proven Delinquency Prevention 
Strategy. 1997, NCJ 164834 (8 pp.). 

Mentoring for Youth in Schools and Communi- 
ties Teleconference (Video). 1997, NCJ 166376 
(120 min.), $17.00. 

Mobilizing Communities To Prevent Juvenile 
Crime. 1997, NCJ 165928 (8 pp.). 

Reaching Out to Youth Out of the Education 
Mainstream. 1997, NCJ 163920 (12 pp.). 
Serious and Violent Juvenile Offenders. 1998, 
NCJ 170027 (8 pp.). 

Treating Serious Anti-Social Behavior in Youth: 
The MST Approach. 1997, NCJ 165151 (8 pp.). 
The Youngest Delinquents: Offenders Under 
Age 15. 1997, NCJ 165256 (12 pp.). 

Gangs 

Gang Members and Delinquent Behavior. 1997, 
NCJ 165154 (6 pp.). 

Youth Gangs: An Overview. 1998, NCJ 167249 

(20 pp.). 

Youth Gangs in America Teleconference 
(Video). 1997, NCJ 164937 (120 min.), $17.00. 

General Juvenile Justice 

Comprehensive Juvenile Justice in State 
Legislatures Teleconference (Video). 1 998, 

NCJ 169593 (120 min.). $17.00. 

Developmental Pathways in Boys’ Disruptive 
and Delinquent Behavior. 1997, NCJ 165692 
(20 pp.). 

Exciting Internships: Work Today fora Better 
Tomorrow. 1998, NCJ 171696 (6 pp.). 
Guidelines for the Screening of Persons Work- 
ing With Children, the Elderly, and Individuals 
With Disabilities in Need of Support. 1 998, 

NCJ 167248 (52 pp.). 

Juvenile Justice, Volume III, Number 2. 1997, 
NCJ 165925 (32 pp.). 

Juvenile Justice, Volume IV, Number 2. 1997, 
NCJ 166823 (28 pp.). 

Juvenile Justice, Volume V, Number 1. 1998, 
NCJ 170025 (32 pp.). 

Juvenile Justice Reform Initiatives in the States 
1994-1996. 1997, NCJ 165697 (81 pp.). 

A Juvenile Justice System for the 21st Century. 
1998, NCJ 169726 (8 pp.). 

Juvenile Offenders and Victims: 1997 Update 
on Violence. 1997, NCJ 165703 (32 pp.). 



Juvenile Offenders and Victims: A National 
Report. 1995, NCJ 153569 (188 pp.). 

Keeping Young People in School: Community 
Programs That Work. 1997, NCJ 162783 
(12 pp.). 

Sharing Information: A Guide to the Family 
Educational Rights and Privacy Act and 
Participation in Juvenile Justice Programs. 

1997, NCJ 163705 (52 pp.). 

Missing and Exploited Children 

Court Appointed Special Advocates: A Voice 
for Abused and Neglected Children in Court. 
1997, NCJ 164512 (4 pp.). 

Federal Resources on Missing and Exploited 
Children: A Directory for Law Enforcement and 
Other Public and Private Agencies. 1997, 

NCJ 168962 (156 pp.). 

In the Wake of Childhood Maltreatment. 1997, 
NCJ 165257 (16 pp.). 

Portable Guides to Investigating Child Abuse: 
An Overview. 1997, NCJ 165153 (8 pp.). 
Protecting Children Online Teleconference 
(Video). 1998, NCJ 170023 (120 min.), $17.00. 
When Your Child Is Missing: A Family Survival 
Guide. 1998, NCJ 170022 (96 pp.). 

Substance Abuse 

Beyond the Bench: How Judges Can Help Re- 
duce Juvenile DUI and Alcohol and Other Drug 
Violations (Video and discussion guide). 1996, 
NCJ 162357 (16 min.), $17.00. 

Capacity Building for Juvenile Substance 
Abuse Treatment. 1997, NCJ 167251 (12 pp.). 
The Coach’s Playbook Against Drugs. 1998, 
NCJ 173393 (20 pp.). 

Drug Identification and Testing in the Juvenile 
Justice System. 1998, NCJ 167889 (92 pp.). 
Juvenile Offenders and Drug Treatment: 
Promising Approaches Teleconference (Video). 
1997, NCJ 168617 (120 min.), $17.00. 
Preventing Drug Abuse Among Youth Telecon- 
ference (Video). 1997, NCJ 165583 (120 min.), 
$17.00. 

Violence and Victimization 

Child Development-Community Policing: 
Partnership in a Climate of Violence. 1997, 

NCJ 164380 (8 pp.). 

Combating Fear and Restoring Safety in 
Schools. 1998, NCJ 167888 (16 pp.). 
Epidemiology of Serious Violence. 1997, 

NCJ 165152 (12 pp.). 

Guide for Implementing the Comprehensive 
Strategy for Serious, Violent, and Chronic 
Juvenile Offenders. 1995, NCJ 153681 
(255 pp.). 

Serious and Violent Juvenile Offenders: Risk 
Factors and Successful Interventions Telecon- 
ference (V\6eo). 1998, NCJ 171286 (120 min.), 
$17.00. 

State Legislative Responses to Violent Juvenile 
Crime: 1996-97 Update. 1998, NCJ 172835 

(16 pp.). 

White House Conference on School Safety: 
Causes and Prevention of Youth Violence 
Teleconference {V\6eo). 1998, NCJ 173399 
(240 min.), $17.00. 

Youth in Action 

Planning a Successful Crime Prevention 
Project. 1998, NCJ 170024 (28 pp.). 







